Thin multi-aperture imaging system with auto-focus and methods for using same

ABSTRACT

Dual-aperture digital cameras with auto-focus (AF) and related methods for obtaining a focused and, optionally optically stabilized color image of an object or scene. A dual-aperture camera includes a first sub-camera having a first optics bloc and a color image sensor for providing a color image, a second sub-camera having a second optics bloc and a clear image sensor for providing a luminance image, the first and second sub-cameras having substantially the same field of view, an AF mechanism coupled mechanically at least to the first optics bloc, and a camera controller coupled to the AF mechanism and to the two image sensors and configured to control the AF mechanism, to calculate a scaling difference and a sharpness difference between the color and luminance images, the scaling and sharpness differences being due to the AF mechanism, and to process the color and luminance images into a fused color image using the calculated differences.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/249,937 filed Jan. 17, 2019, which wascontinuation application of U.S. patent application Ser. No. 15/982,401(issued as U.S. Pat. No. 10,250,797) which was a continuation of U.S.patent application Ser. No. 15/407,271 (issued as U.S. Pat. No.9,998,653), which was a continuation application of U.S. patentapplication Ser. No. 14/906,116 (issued as U.S. Pat. No. 9,571,731),which was a 371 application from international applicationPCT/IB2014/063393 and is related to and claims priority from U.S.Provisional Patent Application No. 61/861,185 filed Aug. 1, 2013 andhaving the same title, which is incorporated herein by reference in itsentirety.

FIELD

Embodiments disclosed herein relate in general to digital cameras and inparticular to thin multi-aperture digital cameras with auto-focus.

BACKGROUND

In recent years, mobile devices such as cell-phones, tablets and laptopshave become ubiquitous. Most of these devices include one or two compactcameras—a main rear-facing camera (i.e. a camera on the back side of thedevice, facing away from the user and often used for casual photography)and a secondary front-facing camera (i.e. a camera located on the frontside of the device and often used for video conferencing).

Although relatively compact in nature, the design of most of thesecameras is very similar to the traditional structure of a digital stillcamera, i.e. they comprise an optical component (or a train of severaloptical elements and a main aperture) placed on top of an image sensor.The optical component (also referred to as “optics”) refracts theincoming light rays and bends them to create an image of a scene on thesensor. The dimensions of these cameras are largely determined by thesize of the sensor and by the height of the optics. These are usuallytied together through the focal length (“f”) of the lens and its fieldof view (FOV)—a lens that has to image a certain FOV on a sensor of acertain size has a specific focal length. Keeping the FOV constant, thelarger the sensor dimensions (e.g. in an X-Y plane), the larger thefocal length and the optics height.

As the dimensions of mobile devices shrink, the compact cameradimensions become more and more a key factor that limits the devicethickness. Several approaches have been proposed to reduce the compactcamera thickness in order to alleviate this constraint. Recently,multi-aperture systems have been proposed for this purpose. In suchsystems, instead of having one aperture with one train of opticalelements, the camera is divided into several apertures, each withdedicated optical elements, all apertures sharing a similar field ofview. Hereinafter, each such aperture, together with the optics and thesensor area on which the image is formed, is defined as a “sub-camera”.Typically, in multi-aperture camera designs, each sub-camera creates asmaller image on the image sensor compared with the image created by areference single-aperture camera. Therefore, the height of eachsub-camera can be smaller than the height of a single-aperture camera,reducing the total height of the camera could be reduced and allowingfor slimmer designs of mobile devices.

FIG. 1A and FIG. 1B show a schematic design of a traditional camera andof a dual-aperture camera with two sub-cameras, respectively. Atraditional camera 100′ in FIG. 1A includes an image sensor 102 placedon a substrate 104 and a lens 106. A “camera height” is defined as theheight of the camera module, from substrate 104 to the top of lens 106.A dual-aperture camera 100″ in FIG. 1B includes two sub-cameras, asub-camera 1 with an image sensor 112 a and a lens 116 a with an opticalaxis 118 a, and a sub-camera 2 with, an image sensor 112 b and a lens116 b with an optical axis 118 b. The two sensors are placed on,respectively, substrates 114 a and 114 b. For comparison's sake, it isassumed that the reference single-aperture camera and the dual-aperturecamera have the same field of view (FOV) and the sensors have the samepixel size. However, image sensor 102 has a higher resolution (number ofpixels) compared with image sensor 112 a or image sensor 112 b, and istherefore larger in size. The potential advantage in camera height ofthe dual-aperture camera (i.e. the thickness from substrate 114 a to thetop of lens 116 a and from substrate 114 b to the top of lens 116 b) maybe appreciated.

There are several significant challenges involved in multi-aperturecamera designs. First and foremost, the sensor area of each sub-camerais smaller compared with that of a single-aperture camera. If the pixelsize in each sub-camera sensor is kept the same as that in thesingle-aperture camera sensor, the resolution of an image captured byeach sub-camera is smaller than that captured by the single-aperturecamera. If the resolution of the output image is to be kept the same,the images from the different sub-cameras need to be combined into ahigher-resolution image. This is usually done in the digital domain, bya dedicated algorithm. Several methods have been proposed for combininglower-resolution images to produce a higher-resolution image. Somealgorithms in such methods require a registration step between the setof low-resolution images, to account for parallax (which is present in amulti-aperture camera system due to the shift in point-of-view betweensub-cameras). One such algorithm is described in co-assigned PCT patentapplication PCT/IB2014/062180 titled “Dual aperture zoom digitalcamera”, which is incorporated herein by reference in its entirety.

Another challenge relates to the requirement that the camera provides anin-focus image for a wide range of object distances (usually fromseveral centimeters to infinity in compact camera modules). To fulfillthis requirement, a single-aperture camera may include an Auto-Focus(AF) mechanism that controls the focus position of the optics, by movingthe optical element along the optical axis, thus changing its heightabove the sensor. In multi-aperture cameras, in order to support anin-focus image for a wide range of object distances, a straightforwardapproach would be to provide a dedicated AF mechanism in eachsub-camera. This approach has several drawbacks including increased sizeand cost of the camera, higher operating power and more complicatedcontrol, as the AF mechanisms of each sub-camera needs to besynchronized, to ensure all of the sub-cameras are focused to the sameposition.

Another complication that may arise when using an AF mechanism in amulti-aperture camera is connected with the algorithm that combines thelower resolution sub-camera images to produce a higher resolution image.Since an AF mechanism moves the optical element along the optical axisabove the sensor, it scales the image that is formed on the sensor tosome extent. Slight differences between the focusing positions ofdifferent AF mechanisms in each sub-camera may result in differentscales applied to the lower resolution sub-camera images. Suchdifferences in scale may degrade the performance of the imageregistration step in the algorithm. Correcting for the different scaleis not trivial, due to the dynamic nature of the scale—the scale appliedon the image depends on the focus position of the optics, which in turnchanges with object distance. This means that the scale cannot betrivially corrected by calibrating the multi-aperture camera andapplying a fixed correction, but rather, the correct scale has to beestimated at each image. Estimating the correct scale to apply from theimage is not trivial, in the presence of parallax (where differentobjects appear at different locations as a function from their distancefrom the camera) and in the presence of possible occlusions of objectsin one aperture but not in the other. There is therefore a need for amethod that can accurately estimate and correct differences in scalingon a per-image basis.

As an alternative to using AF, multi-aperture camera designs have beenproposed with no AF mechanism at all. Such designs rely on the smallerfocal length of each sub-camera to provide increased depth-of-focus(DOF) compared with a corresponding single-aperture camera that supportsa larger sensor. Since a larger DOF means that a wider range of objectdistances is imaged in-focus onto the sensor, the AF mechanism could beremoved. While this approach is advantageous in terms of cost, size andsystem complexity, the larger DOF that results from the shorter focallength of a multi-aperture camera is often insufficient to support anin-focus image for object distances ranging from a few centimeters toinfinity. In these cases, settling for a multi-aperture camera withfixed-focus optics results in poor imaging performance at close objectdistances.

Between using multiple AF mechanisms and using only fixed-focus optics,there is a need for a multi-aperture camera system that combines thebenefits of an AF mechanism without adding additional complexity andcost to the camera system.

SUMMARY

Embodiments disclosed herein provide designs of a multi-aperture camerawith an AF mechanism, describe an algorithm that dynamically correctsfor scale differences between sub-camera images, and propose a colorfilter array (CFA) design that may result in higher resolution andsensitivity when combining sub-camera images, compared with standardCFAs.

In various embodiments, there are provided dual-aperture digital cameraswith auto-focus (AF) for imaging an object or scene, each suchdual-aperture digital camera comprising a first sub-camera that includesa first optics bloc and a color image sensor with a first number ofpixels, the first camera operative to provide a color image of theobject or scene, a second sub-camera that includes a second optics blocand a clear image sensor having a second number of pixels, the secondsub-camera operative to provide a luminance image of the object orscene, the first and second sub-cameras having substantially the samefield of view, an AF mechanism coupled mechanically at least to thefirst optics bloc, and a camera controller coupled to the AF mechanismand to the two image sensors and configured to control the AF mechanism,to calculate a scaling difference and a sharpness difference between thecolor and luminance images, the scaling and sharpness differences beingdue to the AF mechanism, and to process the color and luminance imagesinto a fused color image using the calculated differences.

The first number of pixels and second number of pixels may be equal ordifferent. The first and second images sensors are formed on a singlesubstrate. The first sub-camera may include an infra-red (IR) filterthat blocks IR wavelengths from entering the color image sensor and thesecond sub-camera may be configured to allow at least some IRwavelengths to enter the clear image sensor. In some embodiments, thecolor image sensor may include a non-Bayer color filter array (CFA).

In an embodiment, the AF mechanism may be coupled mechanically to thefirst optics bloc, and the second optics bloc may have a fixed focusposition. In an embodiment, the fixed focus position may be such that aDOF range of the second sub-camera is between infinity and less thanabout 100 cm. In an embodiment, the AF mechanism may be coupledmechanically to the first and second optics blocs and operative to movethem together in a direction common to respective optics bloc opticalaxes.

In an embodiment, the camera may further comprise an optical imagestabilization mechanism coupled mechanically to the first and secondoptics blocs and in a direction perpendicular to respective optics blocoptical axes to optically stabilize the AF fused color image.

In an embodiment there is provided method for obtaining a focused colorimage of an object or scene using a dual-aperture camera, comprising thesteps of obtaining simultaneously an auto-focused color image and anauto-focused or fixed focus luminance image of the object or scene,wherein the color image has a first resolution, a first effectiveresolution and a first signal-to-noise ratio (SNR), and wherein theluminance image has a second resolution, a second effective resolutionand a second SNR, preprocessing the two images to obtain respectiverectified, normalized and scale-adjusted color and luminance imagesconsidering scaling and sharpness differences caused by the AF action,performing local registration between the rectified, normalized andscale-adjusted color and luminance images to obtain registered images,and fusing the registered images into a focused fused color image.

In an embodiment, the step of preprocessing to obtain scale-adjustedcolor and luminance images includes calculating a set of correspondingpoints in the color and luminance images, extracting a single coordinatefrom each corresponding point and using the single coordinate toestimate a scaling factor S between the color and luminance images. Theextracted coordinate is Y and the scaling factor S may be given byS=(Y2′*W*Y2)\Y2′*W*Y1, where Y1 is a vector of Y coordinates of pointstaken from one image, Y2 is a vector of Y coordinates of points takenfrom the other image, and W is a diagonal matrix that holds the absolutevalues of Y2.

In an embodiment, a method may further comprise using scaling factor Sto scale one of the images to match the other image, thereby obtainingthe registered images.

In an embodiment, a method may further comprise optically stabilizingthe obtained color and luminance images.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting examples of embodiments disclosed herein are describedbelow with reference to figures attached hereto that are listedfollowing this paragraph. The drawings and descriptions are meant toilluminate and clarify embodiments disclosed herein, and should not beconsidered limiting in any way.

FIG. 1A shows schematically the design of a traditional digital camera;

FIG. 1B shows schematically the design of a dual-aperture camera;

FIG. 2 shows schematically an embodiment of a dual-aperture imagingsystem with auto-focus disclosed herein, in (a) a general isomeric view,and (b) a sectioned isomeric view;

FIG. 3 shows an embodiment of an image sensor for the imaging system inFIG. 2, in which one sub-camera has a CFA sensor, while anothersub-camera has a clear sensor;

FIG. 4A shows schematically in a flow chart an embodiment of a methoddisclosed herein;

FIG. 4B shows in a flow chart details of the scale adjustment step inthe method shown in FIG. 4A;

FIG. 4C shows two images with corresponding points;

FIG. 5A shows schematically another embodiment of a dual-apertureimaging system with a single auto-focus mechanism disclosed herein in asectioned isomeric view;

FIG. 5B shows schematically in a flow chart an embodiment of a methodfor auto-focus imaging with the imaging system in FIG. 5A;

FIG. 6 shows schematically yet another embodiment of a dual-apertureimaging system numbered with a single auto-focus mechanism in asectioned isomeric view.

DETAILED DESCRIPTION

FIG. 2 shows schematically an embodiment of a dual-aperture imagingsystem with auto-focus disclosed herein and numbered 200, in (a) ageneral isomeric view, and (b) a sectioned isomeric view. In thefollowing description, “imaging system” and “camera” may be usedinterchangeably. System 200 comprises two sub-cameras, labeled 202 and204, each sub-camera having its own optics. Thus, sub-camera 202includes an optics bloc 206 with an aperture 208 and an optical lensmodule 210, as well as a sensor 212. Similarly, sub-camera 204 includesan optics bloc 214 with an aperture 216 and an optical lens module 218,as well as a sensor 220. The sensors are also referred to henceforth as“sensor 1” (212) and “sensor 2” (220). Note that the two sensors may beimplemented as two distinct areas on the same substrate, and notnecessarily as two stand-alone sensors. Each optical lens module mayinclude several lens elements as well as an Infra-Red (IR) filter 222 a,b. In some embodiments, some or all of the lens elements belonging todifferent apertures may be formed on the same substrate. The twosub-cameras are positioned next to each other, with a small baseline 224between the two apertures 208 and 216. Each sub-camera further includesan auto-focus mechanism, respectively 226 and 228.

The sensors used in each sub-camera may have different color filterarrays (CFAs). In some embodiments, sensor 1 may have one type of CFA,while sensor 2 may have another type of CFA. In some embodiments, sensor1 may have a CFA and sensor 2 may have a “white” or “clear” filter array(marked by “W”)—in which all the pixels absorb the same wide range ofwavelengths, e.g. between 400 nm and 700 nm (instead of each pixelabsorbing a smaller portion of the spectrum). A sensor having a colorfilter array may be referred to henceforth as a “color image sensor”,while a sensor with a clear or W filter array is referred to as a “clearimage sensor”. FIG. 3A shows a sensor embodiment 300, where numeral “1”represents sensor 1 (with a CFA) and numeral “2” represents sensor 2(with a clear “white” filter array). Circles 302 a, 302 b mark imagecircles formed by the optics on the sensors, while a white area 304marks the substrate on which the sensors are located. Circles 302 a, 302b may be larger than the respective size of the sensor the image isformed on. In some cases, overlap between the two image circles mayoccur and mechanical light blocking elements (e.g., walls) may be neededto prevent optical cross-talk between the sub-cameras.

The CFA of sensor 1 may be standard or non-standard. As used herein, a“standard CFA” may include a known CFA such as Bayer, RGBE, CYYM, CYGMand different RGBW filters such as RGBW #1, RGBW #2 and RGBW #3. Forexample, non-Bayer CFA patterns include repetitions of a 2×2 micro-cellin which the color filter order is RRBB, RBBR or YCCY whereY=Yellow=Green+Red, C=Cyan=Green+Blue; repetition of a 3×3 micro-cell inwhich the color filter order is GBRRGBBRG (e.g. as in sensor 1 in FIG.3A); and repetitions of a 6×6 micro-cell in which the color filter orderis one of the following options:

-   -   1. Line 1: RBBRRB. Line 2: RWRBWB. Line3: BBRBRR. Line 4:        RRBRBB. Line 5: BWBRWR. Line 6: BRRBBR.    -   2. Line 1: BBGRRG. Line 2: RGRBGB. Line 3: GBRGRB. Line 4:        RRGBBG. Line 5: BGBRGR. Line 6: GRBGBR.    -   3. Line 1: RBBRRB. Line 2: RGRBGB. Line 3: BBRBRR. Line 4:        RRBRBB. Line 5: BGBRGR. Line 6: BRRBBR.    -   4. Line 1: RBRBRB. Line 2: BGBRGR. Line 3: RBRBRB. Line 4:        BRBRBR. Line 5: RGRBGB. Line 6: BRBRBR.        The color CFA of sensor 1 in FIG. 3B is a Bayer pattern. By        using a CFA on sensor 1, sub-camera 1 captures the color        information about the scene, while sub-camera 2 captures        luminance information about the scene.

The CFA pattern of sensor 1 in FIG. 3A as well as other non-Bayer CFAslisted above an advantage over the standard Bayer pattern in that theydivide the red, green and blue colors evenly across the sensor pixels.This results in a finer sampling of red and blue colors, while the greencolor experiences coarser sampling compared with the standard Bayerpattern. However, as the image that is captured by sensor 2 is used toextract luminance information about the scene (instead of relying on thegreen channel for that, as is the case when using a Bayer CFA), thegreen pixels are only used for color information. In traditional compactcamera design, a filter that lets in light in the visible range andblocks light in the IR range is typically placed in the optical path,sometimes as part of a cover glass that protects the sensor. Althoughthe blocking of IR light wastes photons, it allows for a more accurateestimation of the color in the scene, as it reduces color crosstalk fromthe spectral response of the R, G and B color filters (which may besensitive to IR light). In an embodiment, clear sensor 2 is madesensitive to IR light by removing the IR filter or by redesigning itsspectral response to let in some light in the IR range. The motivationfor capturing IR light, in addition to light in the visible range, is toincrease the Signal-to-Noise Ratio (SNR) in the image, as many naturaland artificial light sources also emit photons in the IR spectrum.Unlike a sensor with a color CFA (i.e. sensor 1), absorption of IR lightdoes not introduce color cross-talk in clear sensor 2 (since the sensorrecords a panchromatic image of the scene).

Removing the IR filter may have some negative implications on imagequality. For example, extending the range of wavelengths that arecaptured by the camera may lead to longitudinal chromatic aberrationsthat may degrade the Point Spread Function (PSF), resulting in ablurrier image. To address this issue, in an embodiment, the optics ofsub-camera 2 are optimized across both the visible and the IR range, tomitigate the effect of chromatic aberrations and to result in a morecompact PSF compared with standard compact camera optics that use an IRfilter. This is unlike the standard optimization process, whichconsiders only wavelengths inside the visible range.

In use, the two sub-cameras share a similar FOV and have substantiallyequal (limited only by manufacturing tolerances) focal lengths. An imagecapture process is synchronized, so that the two sub-cameras capture animage of the scene at a particular moment. Due to the small baselinebetween the two apertures (which could be only a few millimeters, forexample 6.5 mm or 8.5 mm) of the sub-cameras, the output images may showparallax, depending on the object distances in the scene. A digitalimage processing algorithm combines the two images into one image, in aprocess called “image fusion”. Henceforth, the algorithm performing thisprocess is called “image fusion algorithm”. The resulting image may havea higher resolution (in terms of image pixels) and/or a higher“effective resolution” (in terms of the ability to resolve spatialfrequencies in the scene, higher “effective resolution” meaning theability to resolve higher spatial frequencies) and/or a higher SNR thanthat of one sub-camera image.

In terms of resolution and exemplarily, if each sub-camera produces a 5megapixel (2592×1944 pixels) image, the image fusion algorithm maycombine the two images to produce one image with 8 megapixel (3264×2448pixels) resolution. In terms of effective resolution, assuming that animaged object or scene includes spatial frequencies, the use of adual-aperture camera having a clear sensor and a color sensor asdisclosed herein leads to an overall increase in effective resolutionbecause of the ability of the clear sensor to resolve higher spatialfrequencies of the luminance component of the scene, compared with acolor sensor. The fusion of the color and clear images as performed in amethod disclosed herein (see below) adds information in spatialfrequencies which are higher than what could be captured by a color(e.g. Bayer) sub-camera.

In order to generate a higher-resolution or higher effective resolutionimage, the image fusion algorithm combines the color information fromsub-camera 1 with the luminance information from sub-camera 2. Sinceclear sensor 2 samples the scene at a higher effective spatial samplingrate compared with any color channel or luminance thereof in the colorsensor 1, the algorithm synthesizes an image that includes informationat higher spatial frequencies compared with the output image fromsub-camera 1 alone. The target of the algorithm is to achieve a spatialresolution similar to that obtained from a single-aperture camera with asensor that has a higher number of pixels. Continuing the example above,the algorithm may combine two 5 megapixel images, one color and oneluminance, to produce one 8 megapixel image with information contentsimilar to that of a single-aperture 8 megapixel color camera.

In addition to improved spatial resolution, the image fusion algorithmuses the luminance information from clear sensor 2 to generate an imagewith increased SNR, vs. an image from a corresponding single-aperturecamera. The fact that the pixels of sensor 2 are not covered by colorfilters allow each pixel to absorb light in a wider wavelength spectrum,resulting in a significant increase in the light efficiency comparedwith a color CFA camera. In an embodiment, the fusion of clear imageinformation and color image information then provides a +3 dB SNRincrease over that of a single aperture digital camera.

As clear sensor 2 is more sensitive than color sensor 1, there may be aneed to adjust exposure times or analog gains to match the digitalsignal levels between the two cameras. This could be achieved by fixingthe same exposure times to both sensors and configuring a differentanalog gain to each sensor, or by fixing the analog gain in both sensorsand configuring a different exposure time to each sensor.

FIG. 4A shows schematically, in a flow chart, an embodiment of a methoddisclosed herein. FIG. 4B shows in a flow chart details of the scaleadjustment step in the method shown in FIG. 4A. Two images 400 a and 400b from respectively sub-cameras 1 and 2 serve as inputs. The two imagesundergo pre-processing, in respectively step 402 a for the color imageof sensor 1 and 402 b for the luminance image of sensor 2. Step 402 aincludes digital image signal processing (ISP) in an ISP pipeline. TheISP generates a full color image, with R, G, B values at each imagepixel. If the CFA pattern on sensor 1 is non-Bayer, the ISP includesnon-standard demosaicing to interpolate the missing colors at each pixellocation. In addition to demosaicing, other standard ISP pipelinealgorithms may be applied on the image, e.g., black level correction,defect pixel correction, noise removal, etc, as known in the art. Theluminance image from sub-camera 2 is also pre-processed to correct fordefects, noise, shading profile, blur and other optical, analog anddigital aberrations. Normalization, rectification and scale adjustmentare then applied on the two images in step 404. First, the two imagesare normalized to have the same mean signal intensity and standarddeviation (which is a measure for the image dynamic range). This is doneby subtracting the mean from each pixel and dividing each pixel by thestandard deviation in each image. Then, the images are rectified byapplying two projection matrices, in order to correct for differentrotations around the x, y and z axes, to correct for x-y translations ofthe optical center of the two cameras and to fix lens distortions. Theprojection matrices parameters are pre-calculated from calibration data,which may be acquired through a calibration step that is applied foreach camera module during camera module assembly. The data may be savedin one-time programmable memory or EEPROM in the camera module. Afterthe rectification step, epipolar lines in both images are more-or-lessparallel to the horizontal axis of the image, in case the twosub-cameras are positioned one beside the other along the X-axis, orparallel to the vertical axis of the image, in case the two sub-camerasare positioned one beside the other along the Y axis.

The scale adjustment, done after the rectification step, is describednow in more detail with reference to FIG. 4B. Preprocessed and rectifiedimages 418 a and 418 b (also shown exemplarily in FIG. 4C) fromrespectively sub-cameras 1 and 2 serve as inputs. In step 420,corresponding points between the two images are found. In an embodiment,the set of corresponding points is calculated over the entire image. Inanother embodiment, the set of corresponding points is found for aspecific region of interest (ROI) in each image. FIG. 4C, which showsschematically two images A and B of the same scene captured by adjacentcameras (i.e. A captured by sub-camera 1 and A′ captured by sub-camera2) with some parallax—due to the different viewpoint, objects are imagedwith some displacement in one image compared with the other, dependingon their distance from the cameras. Pairs of features a-b-b′ and c-c′represent the same “corresponding points” in the two images A and A′. Analgorithm is used to find corresponding points between the two images. Aset of prominent points are found (e.g. corners) in the two images andthen the algorithm finds matches between the points in the two images.Such algorithms are known to the skilled in the art. In step 422, the Ycoordinate only is extracted in order to estimate the scale between thetwo images. Since the position of the optics, which is controlled by theAF mechanism, may introduce different scales between the two sub-cameraimages, the proper scale needs to be determined for each captured image(i.e. for each focus position). Assuming the two sub-cameras arepositioned adjacent to one another along the X-axis, once correspondingpairs of points are found, a single coordinate is extracted from eachpoint in step 422. That is, the algorithm considers only their Ycoordinate and disregards their X coordinate. The inventors haveadvantageously realized that while the X coordinate may be affected byparallax, the Y coordinate is largely unaffected by parallax after therectification step, and therefore the Y coordinates can be used toestimate the scale more robustly. If the two sub-cameras are positionedadjacent along the Y-axis, then once corresponding pairs of point arefound, the algorithm considers only their X coordinate and disregardstheir Y coordinate. Continuing with the assumption of the twosub-cameras being adjacent along the X-axis, the Y coordinates of thecorresponding points are used to estimate a scaling factor S between theimages in step 424. In an exemplary embodiment, the scaling factorestimation is performed using least-squares, in which case S is given byS=(Y2′*W*Y2)\Y2′*W*Y1where Y1 is a vector of Y coordinates of points taken from one image, Y2is a vector of Y coordinates of points taken from the other image, and Wis a diagonal matrix that holds the absolute values of Y2. Scalingfactor S is then used in step 426 to scale one image in order to matchthe scale between the two images. In step 426, point coordinates in eachimage are multiplied by the same scaling factor S. Finally, in step 428,the corresponding pairs of scaled points are used to calculate a shiftin x and y axes between the two images for each axis. In an embodiment,only a subset of the corresponding points that lie in a certain ROI isused to calculate the shift in x and y. For example, the ROI may be theregion used to determine the focus, and may be chosen by the user or thecamera software (SW). The estimated shift is applied on one of theimages or on both images. The result of the scale adjustment process inFIG. 4B (and in step 404, FIG. 4A) are scaled images 430.

Returning now to FIG. 4A, local registration and parallax correction toestimate a disparity map are applied to the scaled images in step 406.The local registration uses scale and shift parameters found in step404. Fusion to enhance the resolution and improve SNR in the final imageis then performed in step 408, by combining information from bothimages, according to the disparity map. The fusion process uses theimage from sub-camera 1 as a baseline. The output is a fused image 410.Post-processing such as tone mapping, gamma correction, contrastenhancement and color correction/enhancement may then be applied to thefused image.

Auto-Focus

As mentioned with respect to FIG. 2, a camera system disclosed hereinincludes an AF mechanism that controls the focus position of the optics.The system shown in FIG. 2 includes two such AF mechanisms. FIG. 5Ashows schematically another embodiment of a dual-aperture imaging systemnumbered 500 with a single auto-focus mechanism in a sectioned isomericview. System 500 includes in addition to the regular image sensors andoptics only one AF mechanism 502, positioned in a color sub-camera 1. Aluminance sub-camera 2 does not have an AF mechanism, being instead afixed-focus camera, with the focus fixed to a certain object distance.The focus position is such that the DOF range of sub-camera 2 is betweeninfinity and several tens of centimeters, depending on the focal lengthand optical design. For example, the DOF may be between infinity and 50cm, such that sub-camera 2 would produce sharp images for objectdistances that lie within this range from the camera. In system 500,sub-camera 1 can produce an image in which the main object is in focusfor a wide range of object distances, so that it appears sharp in asub-camera 1 image, by changing the focus position of the optics.

FIG. 5B shows schematically in a flow chart an embodiment of a methodfor image fusion using an imaging system 500 that has AF. Two images 500a and 500 b from respectively sub-cameras 1 and 2 serve as inputs. Afocus position is chosen for sub-camera 1 in step 502. A check isperformed in step 504 to determine whether the distance of an imagedobject lies within the DOF of sub-camera 2, by calculating a sharpnessmetric on the images of sub-cameras 1 and 2, as known in the art. Thecalculation of the sharpness metric may result in a sharpnessdifference. If the answer in the check of step 504 is “Yes”, the objectwill appear sharp in the sub-camera 2 image. In such a case, imagefusion as described above is applied to the object image obtained byboth sub-cameras in step 506 to achieve higher output resolution andbetter SNR. If the answer to check 504 is “No” (i.e. the object liescloser to the camera, outside the DOF range of sub-camera 2), the objectwill appear blurry (not sharp) in the sub-camera 2 image. In this case,the image from sub-camera 2 is not used to enhance the resolution, butonly to improve the SNR of the image from sub-camera 1. To this end,another algorithm (procedure) similar to the fusion algorithm(procedure) above is applied in step 508. The image from sub-camera 1 isscaled to the proper output size and a de-noising algorithm that usesinformation from the sub-camera 2 image is applied. Since in this casehigh frequencies are lost in the sub-camera 2 image (due to defocus),the algorithm only considers information at low spatial frequencies fromthe image of sub-camera 2. In order to determine the object distance,the chosen focus position of the AF mechanism of sub-camera 1 is used(after the focusing process has converged).

FIG. 6 shows schematically yet another embodiment of a dual-apertureimaging system numbered 600 with a single AF mechanism in a sectionedisomeric view. Similar to system 500, system 600 includes in addition tothe regular image sensors and optics only one AF mechanism 602. However,in contrast with AF mechanism 502, AF mechanism 602 moves the optics ofsub-camera 1 and the optics of sub-camera 2 together. The opticalelements are mounted on a lens holder 604 with dedicated threads to holdthe two lenses, which is moved by the AF mechanism. Since the optics ofsub-camera 1 and sub-camera 2 have very similar focal lengths, themechanical movement brings the image from sub-camera 1 and fromsub-camera 2 to focus at the same time. The advantage of thisconstruction over having only one AF mechanism is that both sub-camerassupport the same range of object distances, so that the image fusionalgorithm can be applied for the entire range. When the AF mechanismchooses the best focus position for the lens, information from bothsub-camera images can be taken into account (e.g. to assist in focusingin low-light situations). In low-light, AF sharpness measurements arenoisier, due to the lower SNR in the images. Using two images instead ofone can help reduce the noise and improve the robustness and accuracy ofthe AF process (algorithm).

In an embodiment, some or all the optical elements of sub-camera 1 andsub-camera 2, are made on the same die, using wafer-level opticsmanufacturing techniques or injection molding of glass or plasticmaterials. In this case, the single AF mechanism moves the optical dieson which the optical elements of the two sub-cameras are fabricated, sothat the two optical stacks move together.

In another embodiment, a camera is similar to camera 500 and includes asingle AF mechanism placed on sub-camera 1 (with the color CFA).Sub-camera 2 does not have an AF mechanism, but uses instead fixed focusoptics with unique characteristics that provide extended depth of focus,which is achieved by means of optical design (e.g., by employing opticswith narrower aperture and higher F-number). The optical performance ofthe optics of sub-camera 2 is designed to support sharp images forobject distances between infinity and several cm from the camera—in thiscase, the fusion algorithm can be applied to enhance output resolutionfor a wider range of object distances compared with the single AFembodiment described above. There is usually a tradeoff between the DOFof the camera and the minimal achievable PSF size across the DOF range.An algorithm may be used to enhance the sharpness of the image capturedby sub-camera 2 before the fusion algorithm is applied to combine thephotos. Such an algorithm is known in the art.

To conclude, dual-aperture cameras and methods of use of such camerasdisclosed herein have a number of advantages over single aperturecameras, in terms of camera height resolution, effective resolution andSNR. In terms of camera height, in one example, a standard 8 Mpix ⅓″camera with a 70 degree diagonal FOV may have a module height of 5.7 mm.In comparison, a dual-aperture camera disclosed herein, with two 5 Mpix¼″ image sensors (one color and one clear), each with 70 degreesdiagonal field of view may have a module height of 4.5 mm. In anotherexample, a standard 8 Mpix ⅓″ camera with a 76 degree diagonal FOV mayhave a module height of 5.2 mm. In comparison, a dual-aperture cameradisclosed herein, with two 5 Mpix ¼″ image sensors (one color and oneclear), each with a 76 degree diagonal FOV, may have a module height of4.1 mm.

While this disclosure has been described in terms of certain embodimentsand generally associated methods, alterations and permutations of theembodiments and methods will be apparent to those skilled in the art.The disclosure is to be understood as not limited by the specificembodiments described herein, but only by the scope of the appendedclaims

What is claimed is:
 1. A multi-aperture imaging system, comprising: a) afirst camera comprising a first sensor with a first number of pixels anda first type of filter array; and b) a second camera comprising a secondsensor with a second number of pixels and a second type of filter arraydifferent from the first type of filter array, wherein the first type offilter array includes repetitions of a 3×3 non-Bayer color filter arrayin the form GBRRGBBRG.
 2. The multi-aperture imaging system of claim 1,wherein the first number of pixels is different from the second numberof pixels.
 3. The multi-aperture imaging system of claim 1, wherein thefirst number of pixels is equal to the second number of pixels.
 4. Themulti-aperture imaging system of claim 2, wherein the first camera andthe second camera have substantially a same field of view.
 5. Themulti-aperture imaging system of claim 2, wherein the second type offilter array includes clear or white filter pixels.
 6. Themulti-aperture imaging system of claim 5, wherein the first camera isoperative to provide a color image of an object or scene, wherein thesecond camera is operative to provide a luminance image of the object orscene, and wherein the multi-aperture imaging system further comprises acamera controller coupled to the two sensors and used to process thecolor image and the luminance image into a fused color image.
 7. Amulti-aperture imaging system, comprising: a) a first camera comprisinga first sensor with a first number of pixels and a first type of filterarray; and b) a second camera comprising a second sensor with a secondnumber of pixels and a second type of filter array different from thefirst type of filter array, wherein the first type of filter array is anon-Bayer color filter that includes a plurality of repetitions, witheach repetition including 16 red color pixel filters, and wherein thefirst type of filter array has, in a first direction, a first red colorpixel filter adjacent to a second red color pixel filter, a third redcolor pixel filter adjacent to a fourth red color pixel filter, a fifthred color pixel filter adjacent to a sixth red color pixel filter and aseventh red color pixel filter adjacent to an eighth red color pixelfilter.
 8. The multi-aperture imaging system of claim 7, wherein thefirst number of pixels is different from the second number of pixels. 9.The multi-aperture imaging system of claim 7, wherein the first numberof pixels is same as the second number of pixels.
 10. The multi-apertureimaging system of claim 8, wherein the second type of filter arrayincludes clear or white pixel filters.
 11. The multi-aperture imagesystem of claim 10, wherein the first camera is operative to provide acolor image of an object or scene, wherein the second camera isoperative to provide a luminance image of the object or scene, andwherein the multi-aperture imaging system further comprises a cameracontroller coupled to the two sensors and used to process the colorimage and the luminance image into a fused color image.
 12. Amulti-aperture imaging system comprising: a) a first camera comprising afirst sensor with a first number of pixels and a first type of filterarray; and b) a second camera comprising a second sensor with a secondnumber of pixels and a second type of filter array different from thefirst type of filter array, wherein the first type of filter array is anon-Bayer color filter that includes a plurality of repetitions, witheach repetition including 16 blue color pixel filters, and wherein thefirst type of filter array has, in a first direction, a first blue colorpixel filter, adjacent to a second blue color pixel filter, a third bluecolor pixel filter adjacent to a fourth blue color pixel filter, a fifthblue color pixel filter adjacent to a sixth blue color pixel filter anda seventh blue color pixel filter adjacent to an eighth blue color pixelfilter.
 13. The multi-aperture imaging system of claim 12, wherein thefirst number of pixels is different from the second number of pixels.14. The multi-aperture imaging system of claim 12, wherein the firstnumber of pixel is same as the second number of pixels.
 15. Themulti-aperture imaging system of claim 13, wherein the second type offilter array includes clear or white pixel filters.
 16. Themulti-aperture imaging system of claim 15, wherein the first camera isoperative to provide a color image of an object or scene, wherein thesecond camera is operative to provide a luminance image of the object orscene, and wherein the multi-aperture imaging system further comprises acamera controller coupled to the two sensors and used to process thecolor image and the luminance image into a fused color image.